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Foreword 


The  United  States  Navy  is  committed  to  providing  the  best  available  environ¬ 
mental  products  to  the  Fleet.  To  this  end,  NORDA  is  comparing  various  opera¬ 
tional  wind  and  wave  models  in  use  around  the  world  to  the  Navy’s  models.  This 
report  describes  such  a  comparison  for  one  day  using  a  wind-  and  wave-measuring 
satellite  as  a  reference. 


A.  C.  Esau,  Captain,  USN 
Commanding  Officer,  NORDA 


Executive  summary 


By  comparing  operational  wind  and  wave  models  to  GEOSAT,  we  found  that 
on  10  March  1986,  the  Federal  Republic  of  Germany  had  the  best  skill  score  for 
a  regional  wind  analysis,  NOAA  had  the  best  score  for  a  global  wind  analysis, 
the  Netherlands  had  the  best  score  for  a  regional  wave  analysis,  and  the  U.S.  Navy 
had  the  best  score  for  a  global  wave  analysis. 
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Ocean  Wind  and  Wave  Model 
Comparisons  with  GEOSAT 


I.  Introduction 

The  U.S.  Navy  is  evaluating  operational  wind  and  wave 
models  to  determine  if  present  Navy  models  need  improve¬ 
ments.  The  evaluation  program  consists  of  intercomparing 
models  by  using  observed  wind  and  wave  fields  recorded  by 
the  GEOdesy  SATellite  (GEOSAT).  The  technique  is  similar 
to  that  reported  by  Cavaleri  et  al.  (1982)  and  Resio  and  Vin¬ 
cent  (1982),  except  that  they  used  hypothetical  wind  and  wave 
fields. 

For  this  evaluation,  we  compared  six  operational  wind  and 
eight  operational  wave  model  analyses  to  GEOSAT  observa¬ 
tions  for  one  day  (10  March  1986).  We  tested  wind  model 
analyses  from  Canada,  the  Netherlands,  the  Federal  Republic 
of  Germany,  Japan,  the  U.S.  Navy,  and  the  U.S.  National 
Oceanic  and  Atmospheric  Administration  (NOAA).  We  tested 
wave  model  analyses  from  Canada  (both  military  and  civilian), 
the  Netherlands,  the  Federal  Republic  of  Germany,  Japan,  the 
U.S.  Navy,  NOAA,  and  a  private  U.S.  company  (Offshore 
and  Coastal  Technology). 

II.  Method 

A.  Model  data 

In  order  to  obtain  coincident  model  output,  we  requested 
that  all  participants  send  us  their  analyzed  (not  forecasted)  wind 
and  wave  fields  for  0000,  0600,  1200,  1800,  and  2400  GMT 
on  10  March  1986.  We  received  the  results  in  a  variety  of 
forms,  such  as  magnetic  tape,  gridded  charts,  and  contoured 
charts.  Next,  we  saved  and  edited  the  GEOSAT  wind  and 
wave  fields.  Finally,  we  matched  the  model  and  GEOSAT  fields 
by  computer  if  we  received  magnetic  tapes,  or  by  hand  if  we 
received  charts. 

B.  GEOSAT  data 

GEOSAT  was  launched  in  March  1985  and  uses  a  radar 
altimeter  to  estimate  wind  speeds  and  wave  heights.  The 
altimeter  is  a  narrow-beam,  downward-looking,  short-pulse 
radar  that  bounces  signals  off  the  ocean’s  surface.  Since  the 
signals  are  stored  aboard  the  satellite  until  it  passes  over  the 
ground  station,  wind  and  wave  data  are  unavailable  from  3 
to  16  hours  after  they  are  sampled.  Table  1  summarizes  the 
satellite’s  characteristics  (Kilgus  et  al.,  1984). 


Table  1 .  GEOSAT  characteristics. 


Orbit: 

Altitude 

Inclination 

Period 

Ground  speed 

Repeat  cycle 

800  km 

108° 

101  min 

6  6  km/sec 

3  Days 

Radar: 

Frequency 

Max.  ocean  spot  size 

Effective  spot  size 

13.5  GHz 

25  km 

2-7  km 

Since  the  satellite  is  in  near-polar  orbit  and  the  reflection 
spot  is  small,  wind  and  wave  coverage  consists  of  narrow  north 
and  south  tracks.  These  tracks  cover  the  earth  at  about  3000 
km  spacing  near  the  equator  and  converge  near  the  poles  (see 

Fig.  1). 

Significant  wave  heights  are  estimated  along  these  tracks 
by  measuring  the  leading-edge  slope  of  the  returning  pulse. 
High  seas  spread  the  return  pulse  and,  hence,  reduce  its  slope. 

Wind  speeds  are  estimated  indirectly.  A  surface  reflection 
coefficient  is  calculated  from  the  return  pulse  magnitude.  The 
pulse  is  absorbed  at  the  ocean  surface  by  capillary  waves  and 
foam,  and  these  factors  depend  on  wind  speed.  Ground  proc¬ 
essing  is  then  used  to  relate  the  reflection  coefficient  to  sur¬ 
face  wind  speed  by  an  empirical  formula.; 

The  satellite  was  designed  to  measure  significant  wave 
heights  within  10%  (waves  greater  than  5  m)  or  0.5  m  (waves 
smaller  than  5  m),  and  to  estimate  wind  speeds  within  1.8 
m/sec  (Kilgus  et  al.,  1984).  To  check  this  accuracy,  we  first 
edited  GEOSAT  data  (to  eliminate  transmission  errors,  islands, 
and  ice),  and  then  matched  times  and  locations  to  the  NOAA 
buoy  network.  Figures  2  and  3  compare  GEOSAT  wind  speeds 
and  wave  heights  to  those  recorded  by  the  NOAA  buoys 
(within  30  min  and  50  km)  during  our  test.  The  mean  buoy 
minus  GEOSAT  wind  difference  was  -0.8  m/sec,  and  the 
standard  deviation  of  the  difference  was  2.2  m/sec.  The  mean 
wave  difference  was  0.1  m  and  the  standard  deviation  was  0.4 
m. 

The  scatter  in  Figures  2  and  3  has  several  sources.  For 
example,  buoys  average  over  time  (8  min  for  wind,  20  min 
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Figure  1.  Track  of  GEOS  AT  on  ocean 's  surface  during  test  day  of  10  March  1986.  Gaps  result  from  editing. 


BUOY  WIND  SPEED  (m/sec) 


BUOY  WAVE  HEIGHT  (m) 


Figure  2.  Comparison  of  wind  speeds  recorded  by  NOAA  buoys  Figure  3.  Comparison  of  significant  wave  heights  recorded  by 
and  GEOS  AT.  Satellite  observations  were  within  30  min  and  25  NOAA  buoys  and  GEOSAT.  Satellite  observations  were  within 
km  of  matching  buoy  observation.  Shaded  band  is  satellite  design  30  min  and  25  km  of  matching  buoy  observation.  Shaded  band 
accuracy.  is  satellite  design  accuracy. 
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for  waves),  and  the  satellite  averages  over  space  (maximum 
of  25  km,  but  most  of  the  highest  energy  return  comes  from 
an  area  2-7  km  from  the  center).  Also,  the  satellite  sensor 
errors  (1.8-m/sec  wind,  0.5-m  wave  height),  shown  as  shaded 
areas  on  the  plots,  are  independent  of  the  buoy  sensor  errors 
(1  m/sec  wind,  0.5  m  wave  height).  All  these  sources  of  scat¬ 
ter  are  mixed  in  Figures  2  and  3-  In  spite  of  these  error  sources, 
however,  these  few  buoy-satellite  comparisons  suggest  that 
GEOSAT  is  useful  for  model  verification. 

C.  Comparison  data 

To  use  the  satellite  data  discussed  above,  we  selected  those 
wind  and  wave  points  on  the  10  March  1986  GEOSAT  tracks 
that  were  within  1.5  hours  of  model  analysis  times.  Next,  if 
we  were  provided  magnetic  tapes,  we  computer  scanned  all 
model  values  for  GEOSAT  values  that  were  within  50  km. 
If  we  were  provided  contour  charts,  the  process  was  more  sub¬ 
jective.  We  plotted  GEOSAT  tracks  on  the  model  output  and 
read  off  overlapping  points. 

The  matched  sets  of  data  pairs  from  the  satellite  and  models 
were  then  arranged  in  tables  by  separating  them  into  three 
classes:  light,  moderate,  and  heavy.  Class  boundaries  were 
selected  so  that  nearly  equal  numbers  of  satellite  observations 
occurred  in  each  class.  This  selection  was  done  to  eliminate 
any  advantage  of  a  model  forecasting  the  most  probable  class. 

Comparing  a  class  row  and  column  sum  in  these  tables  shows 
whether  that  class  occurred  as  often  in  the  model  as  it  was 
observed.  A  column  sum  that  is  larger  than  the  row  sum  for 
the  same  class  indicates  that  the  model  overpredicted  that  class. 
The  opposite  is  also  true:  a  smaller  column  sum,  relative  to 
a  row  sum,  means  that  the  model  underpredicted  that  class 
(Panofsky  and  Brier,  1965). 

An  overall  skill  score  was  also  calculated  from  the  matched 
data  pairs.  To  calculate  this  score,  values  were  combined  into 
a  single  number,  defined  by 

55  =  (R-E)/(T-E)  , 
where 

55  =  skill  score, 

R  =  number  of  times  model  results  agreed  with  GEOSAT 
(sum  of  observations  on  major  diagonal), 

E  =  number  of  times  model  results  agreed  with  GEOSAT 
due  to  chance  (formula  given  below), 

T  =  total  number  of  model  minus  GEOSAT  pairs. 

E  was  calculated  by  multiplying  each  column  sum  by  the 
row  sum  for  that  class,  adding  these  column-row  products, 
and  dividing  by  T. 

The  above  skill  score  ranges  from  1 ,  when  all  model  minus 
GEOSAT  pairs  agree  (all  pairs  fall  on  the  diagonal),  to  0,  when 
the  number  of  pairs  agreeing  is  expected  by  chance. 


Table  2.  Distribution  of  pairs  of  U.S.  Navy  model  (wind  model 
at  top,  wave  model  at  bottom)  and  GEOSAT  satellite  data  for 
1 0  March  1 986,  Data  pairs  are  within  1 .5  hours  and  50  m 
of  each  other.  Table  values  have  been  converted  to 
percentages. 


Navy  global  wind  model  (%) 


Light 

Medium 

Heavy 

Sum 

Light 

36 

7 

4 

47 

Medium 

14 

7 

14 

35 

Heavy 

3 

0 

15 

18 

Sum 

53 

14 

33 

100 

Number  of  observations  =  1 38 


Skill  score  *  0.35 

Light  =  0  to  6  m/s  Medium  =  >6  to  10  m/s  Heavy  =  >10  m/s 


Navy  global  wave  model  (%) 


Light 

Medium 

Heavy 

Sum 

Light 

27 

11 

1 

39 

Medium 

12 

17 

7 

36 

Heavy 

2 

2 

21 

25 

Sum 

41 

i  30 

29 

100 

Number  of  observations  =  243 


Skill  score  =  0.48 

Light  *  0  to  2  m  Medium  *  >  2  to  3  m  Heavy  =  >  3  m 


III.  Results 

A  summary  of  the  wind  model  results  is  shown  in  Table 
3  (units  are  meters  per  second),  and  a  summary  of  the  wave 
model  results  are  shown  in  Table  4  (units  are  meters).  The 
left  side  of  each  table  shows  regional  models,  and  the  right 
side  shows  global  models.  In  the  bottom  row,  as  a  reference 
for  the  model  minus  GEOSAT  statistics,  are  the  buoy  minus 
GEOSAT  means  and  standard  deviations. 

Each  model  entry  covers  one  row  in  the  table.  The  row 
contains  the  mean  difference  of  the  matching  data  pairs  (model 
minus  GEOSAT),  the  standard  deviation  of  these  differences, 
the  number  of  data  pairs,  and  the  skill  score  calculated  by  the 
formula  given  in  Table  4. 

To  estimate  variability  in  Tables  3  and  4,  we  ran  Navy  wind 
and  wave  model  comparisons  with  GEOSAT  on  two  other 
days.  We  found  day-to-day  variability  on  the  order  of  ±0.1 
for  means,  ±0.3  for  standard  deviations,  and  ±0.1  for  skill 
scores. 

The  Federal  Republic  of  Germany  had  the  best  skill  score 
for  a  regional  (North  Atlantic)  wind  chart,  NOAA  had  the 
best  score  for  a  global  wind  chart,  the  Netherlands  had  the 
best  score  for  a  regional  (North  Atlantic)  wave  chart,  the  U.S. 
Navy  had  the  best  score  for  a  global  wave  chart. 


Table  3.  Differences  between  wind  models  and  GEOSAT  data  on  10  March  1986.  The  statistics  of  the  differences  and 
the  skill  score  (defined  in  text)  are  given  for  both  regional  and  global  models.  Differences  between  NOAA  buoys  and  GEOSAT 
are  listed  in  the  bottom  row. 


Regional 

Global 

Mean 

(m/s) 

St,  Dev 
(m/s) 

Number  of 
Observations 

Skill 

Score 

Mean 

(m/s) 

St.  Dev. 
(m/s) 

Number  of 
Observations 

Skill 

Score 

Canadian 

Civilian  II 

0.2 

3.1 

48 

0  33 

Canadian 

Civilian  1 

1,1 

3.4 

47 

0.33 

Canadian 

Military 

Netherlands 

2.2 

2.8 

87 

0.22 

Fed.  Rep. 
Germany 

1  1 

3.7 

62 

0.37 

Japanese 

1.0 

2.4 

14 

★ 

U  S.  Navy 

1.7 

3.6 

66 

0.32 

1.1 

3.9 

138 

0.35 

U.S.  NOAA 

0.1 

3.1 

71 

0.32 

-1.1 

2.8 

74 

0  50 

OCTI 

Buoys 

-0.8 

2.2 

8 

★ 

*  Insufficient  data  to  calculate  skill  score 


Table  4.  Differences  between  wave  models  and  GEOSAT  data  on  10  March  1986.  The  statistics  of  the  differences  and 
the  skill  score  (defined  in  text)  are  given  for  both  regional  and  global  models.  Differences  between  NOAA  buoys  and  GEOSAT 
are  listed  in  the  bottom  row. 


Regional 

Global 

Mean 

(m/s) 

St.  Dev. 
(m/s) 

Number  of 
Observations 

Skill 

Score 

Mean 

(m/s) 

St.  Dev. 
(m/s) 

Number  of 
Observations 

Skill 

Score 

Canadian 

Civilian  II 

0.1 

1.3 

48 

0.53 

Canadian 

Civilian  1 

1.8 

1  6 

38 

V  0,00 

Canadian 

Military 

-0  2 

1.5 

67 

0.31 

Netherlands 

0.0 

11 

87 

0.58 

Fed.  Rep. 
Germany 

0.5 

1.0 

62 

0.29 

Japanese 

0.3 

0.4 

14 

* 

U.S.  Navy 

0.4 

0.7 

81 

0.56 

0.2 

0.8 

243 

0.48 

U.S.  NOAA 

-0.7 

1.2 

129 

0.16 

-0.1 

0.9 

393 

0.33 

OCTI 

-0  1 

1.3 

58 

0.42 

Buoys 

0  1 

0.4 

6 

* 

*  Insufficient  data  to  calculate  skill  score 


IV.  Conclusions 

Our  first  conclusion  is  that  satellites  are  the  ideal  way  to 
validate  wind  and  wave  models.  When  GEOSAT  data  are  prop¬ 
erly  edited,  the  satellite  provides  reasonably  accurate  global 
data.  In  addition,  new  satellites  will  soon  be  launched  that  will 
scan  side  to  side,  so  that  the  whole  ocean  surface  is  covered 
instead  of  scanning  a  narrow  path  as  GEOSAT  does.  As  a 
result  of  such  data,  all  wind  and  wave  models  should  improve. 

Our  second  conclusion  is  that,  considering  how  little  data 
are  routinely  available  to  these  operational  models,  they  did 
fairly  well.  Most  model  differences  from  GEOSAT  were  not 
that  much  larger  than  the  buoy  differences  from  GEOSAT. 

Third,  we  think  that  model  minus  satellite  comparisons  such 
as  those  presented  here  should  be  automated  and  run  routinely. 
This  procedure  would  provide  an  operational  method  for  con¬ 
tinually  testing  relative  model  performance,  or  for  evaluating 
new  models  or  data  assimilation  techniques. 

Fourth,  from  our  ratings  no  clear  relationship  between 
advanced  model  physics  and  operational  performance  can  be 
established.  The  Navy  wave  model,  for  example,  does  not  use 
the  latest  physics,  yet  it  had  a  high  skill  score.  These  models 
are  probably  data  limited  rather  than  physics  limited. 


Finally,  we  plan  to  repeat  this  test  on  another  day.  A  repeat 
test  will  enable  us  to  see  if  the  results  in  Tables  3  and  4  are 
consistent  or  if  some  models  perform  better  in  certain  seasons. 
Since  GEOSAT  will  be  turned  off  and  moved  to  another  orbit 
during  October  and  November  1986,  we  hope  to  do  a  repeat 
test  soon. 
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